Image processing device, imaging device, image processing method, and image processing program

ABSTRACT

Disclosed are an image processing device, an imaging device, an image processing method, and an image processing program capable of, when recovering a deteriorated image due to a point spread function of an optical system, effectively performing phase recovery and suppressing the occurrence of artifact due to frequency recovery processing. The image processing device includes a phase recovery processing unit which subjects image data acquired from an imaging element by capturing an object image using an optical system to phase recovery processing using a phase recovery filter based on a point spread function of the optical system, a gradation correction processing unit which subjects image data subjected to the phase recovery processing to nonlinear gradation correction, and a frequency recovery processing unit which subjects image data subjected to the gradation correction to frequency recovery processing using a frequency recovery filter based on the point spread function of the optical system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of PCT International Application No.PCT/JP2015/053654 filed on Feb. 10, 2015, which claims priority under 35U.S.C § 119(a) to Patent Application No. 2014-071463 filed in Japan onMar. 31, 2014, all of which are hereby expressly incorporated byreference into the present application.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device, an imagingdevice, an image processing method, and non-transitory computer readablerecording medium storing an image processing program regardingrestoration processing based on a point spread function.

2. Description of the Related Art

In an object image captured through an optical system, a point spreadphenomenon in which a point object has minute spread due to theinfluence of diffraction, aberration, or the like caused by the opticalsystem may be observed. A function representing a response to a pointlight source of an optical system is called a point spread function(PSF), and is known as a characteristic responsible for resolutiondeterioration (blurring) of a captured image.

The image quality of the captured image with deteriorated image qualitydue to the point spread phenomenon can be recovered through point imagerestoration processing based on the PSF. The point image restorationprocessing is processing in which deterioration characteristics (pointimage characteristics) caused by aberration or the like of a lens(optical system) are determined in advance, and point spread of thecaptured image is cancelled or reduced through image processing using arestoration filter (recovery filter) according to the point imagecharacteristics.

The point image restoration processing can be roughly classified intofrequency recovery (correction) processing and phase recovery(correction) processing. The frequency recovery processing equalizes,that is, recovers modulation transfer function (MTF) characteristicsdeteriorated by an optical system, and the phase recovery processingequalizes, that is, recovers phase transfer function (PTF)characteristics deteriorated by an optical system.

Intuitively, the phase recovery processing moves an image in afrequency-dependent manner so as to return an asymmetrical PSF shape toas symmetrical a shape as possible.

While the frequency recovery processing and the phase recoveryprocessing can be applied simultaneously as signal processing, eithercorrection may be performed by changing a design method of a filtercoefficient.

Various methods for this kind of point image restoration processing havebeen suggested.

JP2011-124692A discloses an image processing device which, in order tosuppress amplification of in noise due to image recovery processing, orthe like, performs recovery processing on an amplitude component and aphase component of an input image to generate a first image, performsrecovery processing only on the phase component of the input image togenerate a second image, acquires difference information between thegenerated first and second images, and synthesizes the differenceinformation with the second image according to an appropriate recoveryintensity adjustment coefficient to generate a recovery adjusted image.

JP2011-059813A discloses an image processing device which performs imagerecovery for an image subjected to nonlinear correction using blinddeconvolution. This image processing device comprises a correction unitwhich performs correction for reducing nonlinear gradation correctionfor a captured image subjected to nonlinear gradation correction, and animage recovery unit which performs image recovery by applying blinddeconvolution to the captured image with reduced gradation correction.

JP2013-020610A discloses an image processing device which reducesover-recovery of image data by image recovery processing. In this imageprocessing device, image recovery processing is performed for colorimage data in an RGB format before gamma processing, the differencebetween amplification and attenuation of a pixel signal value by gammacorrection is absorbed, and a limit value of a variation is calculatedsuch that the maximum value of the variation of the pixel signal valuebecomes constant even after gamma correction. With this, the technicalproblems in that “a situation in which a deterioration state of imagedata actually obtained does not match a deterioration state of imagedata to be recovered by an image recovery filter occurs due to asaturated pixel”, and “image quality deterioration, such as undershootor overshoot, occurs in an edge portion, and in particular, undershootin a low brightness portion is amplified by gamma processing after imagerecovery processing” have been solved.

The point spread function of the optical system is used in a restorationtechnique of an image with an extended depth of focus, andJP2012-049759A discloses an imaging module which executes imagerestoration in a short period of time with excellent accuracy. In thisimaging module, restoration processing is applied to a brightness signalafter synchronization processing (demosaic processing), whereby it isnot necessary to separately provide parameters of the restorationprocessing for RGB, and it is possible to accelerate the restorationprocessing. Furthermore, adjacent pixels are put together in apredetermined unit and the common restoration processing parameters areapplied to this unit to perform a deconvolution processing, whereby theaccuracy of the restoration processing is improved.

SUMMARY OF THE INVENTION

The above-described point image restoration processing is processing forrestoring an image blurred due to the point spread phenomenon (opticalcharacteristics) by the optical system, to an original sharp image, andis a technique for acquiring a recovered image with image qualitydeterioration eliminated or improved by applying the restoration filterbased on the point spread function to source image data withdeteriorated image quality.

Accordingly, in order to obtain a recovered image in which an objectimage is faithfully reproduced, “assumed image quality deteriorationcharacteristics of the restoration filter” for use in the point imagerestoration processing needs to appropriately match “actual imagequality deterioration characteristics of source image data”.

That is, if image quality deterioration caused by an optical system isexactly ascertained, a restoration filter capable of strictlyeliminating such image quality deterioration is designed, and imagequality deterioration (point spread phenomenon) due to the opticalsystem is accurately reflected in source image data, in principle, it ispossible to obtain “a high-quality image in which an object image isfaithfully reproduced” from “a captured image with deteriorated imagequality”.

However, “the characteristics of the restoration filter” may notappropriately match “image quality deterioration of source image data”depending on the characteristics of the object image or imagingequipment.

For example, source image data fluctuates in image quality depending onthe imaging ability of the imaging element, and in a case where theobject image is very bright, a pixel saturation phenomenon may occur inthe imaging element. In a case where pixel saturation occurs, an imagewaveform profile in this saturated portion is clipped or the like; thus,the obtained source image data is not always subjected to the imagequality deterioration characteristics of the lens optical systemfaithfully.

In this way, source image data to be subjected to the restorationprocessing is affected by not only the deterioration characteristicsresulting from the optical system but also the nonlinear deteriorationcharacteristics resulting from the imaging element or pre-stagenonlinear signal processing, and in particular, in a case where thecontrast of the object image is high, unexpected image qualitydeterioration may occur.

Accordingly, even if the characteristics of the optical system aresufficiently analyzed and a restoration filter capable of suppressingthe influence of the point spread phenomenon is designed, “thecharacteristics of the restoration filter” may not appropriately match“image quality deterioration of source image data” depending on theobject image.

If the restoration processing is performed under conditions in which“the characteristics of the restoration filter” may not appropriatelymatch “image quality deterioration of source image data”, image qualitydeterioration is not sufficiently eliminated, and a high-qualityrecovered image is not obtained. In some cases, image qualitydeterioration is promoted, and ringing or the like is conspicuous in arecovered image.

The degree of image quality deterioration (ringing) occurring in therecovered image depends on various factors. For example, the imagequality of the recovered image after the point image restorationprocessing fluctuates due to the influence of the characteristics of therestoration filter for use in the restoration processing, the datacharacteristics of source image data, to which the restorationprocessing is applied, or other kinds of image processing performedbefore and after the restoration processing. Accordingly, in order tomore effectively prevent or reduce image quality deterioration in therecovered image, a restoration processing method integrally inconsideration of various characteristics is required. In particular, ina case where various object images are captured, image datacharacteristics to be subjected to the restoration processing is notconstant, and images having various characteristics, such as an imagehaving high contrast as a whole or locally, a color-shifted image, andan image with some pixel values in a saturated state, will be subjectedto the restoration processing. Therefore, a restoration processingmethod which is excellent in image toughness to flexibly cope with animage to be processed having various characteristics is required.

On the other hand, in the frequency recovery (correction) processing, inmany cases, the degree of correction for an edge portion of an image islarge and overshoot and/or undershoot occurs near the edge slightly incontrast to the phase recovery (correction) processing. The reasons forthis are as follows.

(1) Divergence of frequency characteristics due to deviation of imagingconditions, such as a lens manufacturing variation and an objectdistance, from assumed imaging conditions.

(2) Frequency characteristics change in a pre-stage of recoveryprocessing with linearity of an imaging element or (in a case of acircuit configuration in which a recovery filter processing is performedafter demosaicing processing) since nonlinear processing, such asdemosaicing processing, is performed.

(3) The number of taps of a recovery filter is insufficient, andfrequency characteristics of the recovery filter diverge from desiredfrequency characteristics.

In many case, while overshoot/undershoot occurring near the edge due tothe frequency recovery processing described above is inconspicuousalone, in a case where gamma correction is performed in a post-stage, ingeneral, since the gamma correction greatly enhances the amplitude ofthe low brightness side, there is a problem in that overshoot/undershootbecomes conspicuous artifact.

If the phase recovery processing is performed after the gammacorrection, a correction effect may be weakened with change in frequencycharacteristics of an image due to the gamma correction.

The inventions described in JP2011-124692A, JP2011-059813A,JP2013-020610A, and JP2012-049759A cannot solve the above-describedproblems, and in JP2011-124692A, JP2011-059813A, JP2013-020610A, andJP2012-049759A described above, there is no description of theabove-described problems, and there is no suggestion relating to “animage processing method which integrally considers various factors inthe processing before and after the restoration processing as well asthe restoration processing itself in the restoration processing usingthe point spread function and is excellent in image toughness toflexibly cope with a source image having various characteristics”.

The invention has been accomplished in consideration of such asituation, and an object of the invention is to provide an imageprocessing device, an imaging device, an image processing method, andnon-transitory computer readable recording medium storing an imageprocessing program capable of, when recovering a deteriorated image dueto a point spread function of an optical system, effectively performingphase recovery and suppressing the occurrence of artifact due tofrequency recovery processing.

In order to attain the above-described object, an image processingdevice according to an aspect of the invention comprises a phaserecovery processing unit which subjects image data acquired from animaging element by capturing an object image using an optical system tophase recovery processing using a phase recovery filter based on a pointspread function of the optical system, a gradation correction processingunit which subjects image data subjected to the phase recoveryprocessing to nonlinear gradation correction, and a frequency recoveryprocessing unit which subjects image data subjected to the gradationcorrection to frequency recovery processing using a frequency recoveryfilter based on the point spread function of the optical system.

According to the aspect of the invention, two kinds of recoveryprocessing including the phase recovery processing using the phaserecovery filter based on the point spread function of the optical systemand the frequency recovery processing using the frequency recoveryfilter based on the point spread function of the optical system areexecuted in two steps for image data acquired from the imaging element.In particular, the phase recovery processing is performed before thegradation correction for gradation-correcting image data nonlinearly,and the frequency recovery processing is performed after the phaserecovery processing and the gradation correction.

Ideally, it is preferable that the frequency recovery processing and thephase recovery processing are performed before the nonlinear gradationcorrection. The reason for this is that the frequency characteristics ofthe image changes nonlinearly with the nonlinear gradation correction,and thus, theoretically, if the restoration processing is not performedbefore the gradation correction, accurate correction cannot be executed.

In the invention, of the frequency recovery processing and the phaserecovery processing, the phase recovery processing is performed beforethe gradation correction, and the frequency recovery processing isperformed after the gradation correction. Since the phase recoveryprocessing is performed before the gradation correction (before thefrequency characteristic of the image changes), it is possible toeffectively perform phase recovery. Furthermore, since the frequencyrecovery processing is performed after the gradation correction,overshoot/undershoot slightly occurring due to the frequency recoveryprocessing is not amplified (enhanced) by the gradation correction, andit is possible to prevent the occurrence of strong artifact.

It is preferable that the image processing device according to anotheraspect of the invention further comprises a storage unit which storesthe phase recovery filter and the frequency recovery filter, the phaserecovery processing unit reads the phase recovery filter from thestorage unit and uses the phase recovery filter in the phase recoveryprocessing, and the frequency recovery processing unit reads thefrequency recovery filter from the storage unit and uses the frequencyrecovery filter in the frequency recovery processing. The phase recoveryfilter and the frequency recovery filter are stored in the storage unit,whereby it is possible to reduce computational costs for generating aphase recovery filter and a frequency recovery filter during recoveryprocessing.

The image processing device according to still another aspect of theinvention may further comprise a storage unit which stores the pointspread function of the optical system, an optical transfer functionobtained by Fourier-transforming the point spread function, or amodulation transfer function indicating an amplitude component of theoptical transfer function and a phase transfer function indicating aphase component of the optical transfer function, the phase recoveryprocessing unit may read the point spread function, the optical transferfunction, or the phase transfer function from the storage unit, maygenerate the phase recovery filter, and may use the generated phaserecovery filter in the phase recovery processing, and the frequencyrecovery processing unit may read the point spread function, the opticaltransfer function, or the modulation transfer function from the storageunit, may generate the frequency recovery filter, and may use thegenerated frequency recovery filter in the frequency recoveryprocessing.

In the image processing device according to still another aspect of theinvention, it is preferable that the phase recovery processing unitsubjects image data acquired from the imaging element, which is imagedata for each color channel, to phase recovery processing using a phaserecovery filter, and the frequency recovery processing unit subjects theimage data subjected to the gradation correction, which is image datafor each color channel, to frequency recovery processing using afrequency recovery filter.

According to still another aspect of the invention, it is possible toperform the phase recovery processing reflecting the phase transferfunction (PTF) of each color channel, thus, to correct various chromaticaberrations, such as chromatic aberration of magnification and axialchromatic aberration, and to perform the frequency recovery processingreflecting the modulation transfer function (MTF) of each color channel.

In the image processing device according to still another aspect of theinvention, it is preferable that the phase recovery processing unitsubjects image data acquired from the imaging element, which is imagedata for each color channel, to phase recovery processing using a phaserecovery filter, and the frequency recovery processing unit subjectsimage data subjected to gradation correction by the gradation correctionprocessing unit, which is image data indicating a brightness componentgenerated from image data for each color channel, to frequency recoveryprocessing using the frequency recovery filter.

According to still another aspect of the invention, it is possible toperform the phase recovery processing reflecting the PTF of each colorchannel and to perform the frequency recovery processing for image data(image data of one channel) indicating the brightness component, wherebyit is possible to reduce computational costs (circuit scale) with adecrease in the number of channels.

In the image processing device according to still another aspect of theinvention, it is preferable that the phase recovery processing unitsubjects image data acquired from the imaging element, which is imagedata indicating a brightness component generated from image data foreach color channel, to phase recovery processing using the phaserecovery filter, and the frequency recovery processing unit subjects theimage data subjected to the gradation correction, which is image dataindicating a brightness component generated from image data for eachcolor channel, to frequency recovery processing using the frequencyrecovery filter. Since the phase recovery processing and the frequencyrecovery processing are respectively performed for image data indicatingthe brightness component of one channel, it is possible to minimizecomputational costs (circuit scale).

It is preferable that the image processing device according to stillanother aspect of the invention further comprises a brightness datageneration unit which generates brightness data indicating a brightnesscomponent from image data for each color channel acquired from theimaging element, the phase recovery processing unit subjects brightnessdata generated by the brightness data generation unit to phase recoveryprocessing using the phase recovery filter, the gradation correctionprocessing unit subjects the brightness data subjected to the phaserecovery processing to nonlinear gradation correction, and the frequencyrecovery processing unit subjects the brightness data subjected to thegradation correction to frequency recovery processing using thefrequency recovery filter.

In the image processing device according to still another aspect of theinvention, the gradation correction processing unit is a gammacorrection processing unit which subjects the image data to gradationcorrection by logarithmic processing. The “logarithmic processing” usedherein is processing for converting data expressed by antilogarithm todata expressed by logarithm, and in this application, further includes,for example, gamma correction processing or the like which is performedfor image data. That is, the “logarithmic processing” also indicatesthat image data is converted to image data expressed by logarithm andthe gamma correction processing as one gradation processing is performedfor image data.

In the image processing device according to still another aspect of theinvention, it is preferable that the bit length of the image datasubjected to phase recovery processing by the phase recovery processingunit is greater than the bit length of the image data subjected tofrequency recovery processing by the frequency recovery processing unit.If the bit length of image data is large, it is possible to performimage processing with higher accuracy, and in particular, if the bitlength of image data before the nonlinear gradation correction is large,it is possible to perform the gradation correction more smoothly duringthe gradation correction.

In the image processing device according to still another aspect of theinvention, the optical system has a lens unit which enlarges a depth offield by modulating a phase. According to this aspect, for image dataobtained through a so-called extended depth of field (focus) (EDoF)optical system, it is possible to perform the restoration processingbased on the point spread function with excellent accuracy. A method(optical phase modulation means) of modulating the phase in the lensunit is not particularly limited, and a phase modulation unit may beprovided between lenses, or a phase modulation function may be providedto a lens itself (for example, an incident surface and/or an outputsurface of the lens).

An imaging device according to still another aspect of the inventioncomprises an imaging element which outputs image data by capturing anobject image using an optical system, and the above-described imageprocessing device.

An image processing method according to still another aspect of theinvention comprises a step of subjecting image data acquired from animaging element by capturing an object image using an optical system tophase recovery processing using a phase recovery filter based on a pointspread function of the optical system, a step of subjecting image datasubjected to the phase recovery processing to nonlinear gradationcorrection, and a step of subjecting image data subjected to thegradation correction to frequency recovery processing using a frequencyrecovery filter based on the point spread function of the opticalsystem.

Non-transitory computer readable recording medium storing an imageprocessing program according to still another aspect of the inventioncauses a computer to execute a step of subjecting image data acquiredfrom an imaging element by capturing an object image using an opticalsystem to phase recovery processing using a phase recovery filter basedon a point spread function of the optical system, a step of subjectingimage data subjected to the phase recovery processing to nonlineargradation correction, and a step of subjecting image data subjected tothe gradation correction to frequency recovery processing using afrequency recovery filter based on the point spread function of theoptical system.

According to the invention, since the phase recovery processing usingthe phase recovery filter based on the point spread function of theoptical system and the frequency recovery processing using the frequencyrecovery filter based on the point spread function of the optical systemare executed in steps for image data acquired from the imaging element,and in particular, the phase recovery processing is performed before thegradation correction for gradation-correcting image data nonlinearly(before the frequency characteristics of the image change), it ispossible to effectively perform phase recovery. Furthermore, since thefrequency recovery processing is performed after the gradationcorrection, overshoot and/or undershoot slightly occurring due to thefrequency recovery processing is not amplified (enhanced) by thegradation correction, and it is possible to prevent the occurrence ofstrong artifact.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a digital camera which is connected toa computer.

FIG. 2 is a block diagram showing a functional configuration example ofa camera body controller.

FIG. 3 is a diagram showing an outline from image capturing to pointimage restoration processing.

FIG. 4 is a diagram showing an example of a change in image quality ofan edge portion (image boundary portion) in an object image, and shows acase where ideal point image restoration processing (no saturation of apixel value and no clipping) is performed.

FIG. 5 is a diagram showing an example of source image data, recoveredimage data, and image data after gradation correction processing in acase where “actual image deterioration characteristics (image blurcharacteristics)” do not completely match “a point spread function asthe basis of a restoration filter to be used”.

FIG. 6 is a diagram showing an example of change in contrast of an edgeportion in an object image in actual point image restoration processing(with saturation of a pixel value and clipping), (a) of FIG. 6 showscontrast inherent in the object image, (b) of FIG. 6 shows contrast insource image data before point image restoration processing, and (c) ofFIG. 6 shows contrast in recovered image data after point imagerestoration processing.

FIG. 7 is a diagram (graph) showing an example of the relationshipbetween pre-processing data and post-processing data by gamma processing(gradation processing in logarithmic processing).

FIG. 8 is a block diagram showing a first embodiment of an imageprocessing unit as an image processing device according to theinvention.

FIG. 9 is a block diagram showing an embodiment of a phase recoveryprocessing unit in the image processing unit.

FIG. 10 is a graph showing an example of input/output characteristics(gamma characteristics) subjected to gradation correction by a gradationcorrection processing unit in the image processing unit.

FIG. 11 is a block diagram showing an embodiment of a frequency recoveryprocessing unit in the image processing unit.

FIG. 12 is a block diagram showing a second embodiment of an imageprocessing unit as an image processing device according to theinvention.

FIG. 13 is a block diagram showing another embodiment of a frequencyrecovery processing unit in the image processing unit.

FIG. 14 is a block diagram showing a third embodiment of an imageprocessing unit as an image processing device according to theinvention.

FIG. 15 is a block diagram showing another embodiment of a phaserecovery processing unit in the image processing unit.

FIG. 16 is a block diagram showing a form of an imaging modulecomprising an EDoF optical system.

FIG. 17 is a diagram showing an example of the EDoF optical system.

FIG. 18 is a diagram showing a restoration example of an image acquiredthrough the EDoF optical system.

FIG. 19 shows the appearance of a smartphone which is an embodiment ofan imaging device of the invention.

FIG. 20 is a block diagram showing the configuration of the smartphoneshown in FIG. 19.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of the invention will be described referring to theaccompanying drawings. In the following embodiment, as an example, acase where the invention is applied to a digital camera (imaging device)which is connectable to a computer (personal computer (PC)) will bedescribed.

FIG. 1 is a block diagram showing a digital camera which is connected toa computer.

A digital camera 10 comprises an interchangeable lens unit 12, and acamera body 14 provided with an imaging element 26, and the lens unit 12and the camera body 14 are electrically connected through a lens unitinput/output unit 22 of the lens unit 12 and a camera body input/outputunit 30 of the camera body 14.

The lens unit 12 is provided with an optical system, such as a lens 16or a diaphragm 17, and an optical system operating unit 18 whichcontrols the optical system. The optical system operating unit 18includes a lens unit controller 20 which is connected to the lens unitinput/output unit 22, and an actuator (not shown) which operates theoptical system. The lens unit controller 20 controls the optical systemthrough an actuator based on a control signal sent from the camera body14 through the lens unit input/output unit 22, and performs, forexample, focus control or zoom control by lens movement, diaphragmamount control of the diaphragm 17, and the like.

The imaging element 26 of the camera body 14 has a condensing microlens,a color filter of R (red), G (green), B (blue), or the like, and animage sensor (a photodiode: a complementary metal oxide semiconductor(CMOS), a charge coupled device (CCD), or the like). The imaging element26 converts light of an object image emitted through the optical system(the lens 16, the diaphragm 17, or the like) of the lens unit 12 to anelectrical signal, and sends an image signal (source image data) to thecamera body controller 28.

The imaging element 26 of this example outputs source image data throughimaging of the object image using the optical system, and source imagedata is transmitted to an image processing unit of the camera bodycontroller 28.

As shown in FIG. 2, the camera body controller 28 has a device controlunit 34 and an image processing unit (image processing device) 35, andintegrally controls the camera body 14. For example, the device controlunit 34 controls the output of the image signal (image data) from theimaging element 26, generates a control signal for controlling the lensunit 12 and transmits the control signal to the lens unit 12 (lens unitcontroller 20) through the camera body input/output unit 30, andtransmits image data (RAW data, JPEG data, and the like) before andafter image processing to external devices (a computer 60 and the like)connected through an input/output interface 32. The device control unit34 appropriately controls various devices, such as a display unit (notshown) (an electronic view finder (EVF) or a rear liquid crystal displayunit), in the digital camera 10.

The image processing unit 35 can subject an image signal from theimaging element 26 to arbitrary image processing as necessary. Forexample, various kinds of image processing, such as sensor correctionprocessing, demosaic (synchronization) processing, pixel interpolationprocessing, color correction processing (offset correction processing,white balance processing, color matrix processing, gradation correctionprocessing (gradation correction processing unit 33), and the like), RGBimage processing (sharpness processing, tone correction processing,exposure correction processing, contour correction processing, and thelike), RGB/YCrCb conversion processing, and image compressionprocessing, are appropriately performed in the image processing unit 35.In particular, the image processing unit 35 of this example comprises apoint image restoration control processing unit 36 which subjects animage signal (source image data) to restoration processing (point imagerestoration processing) based on a point spread function of the opticalsystem. The details of the point image restoration processing will bedescribed below.

The digital camera 10 shown in FIG. 1 is provided with other devices (ashutter and the like) necessary for imaging or the like, and the usercan appropriately determine and change various settings (exposure value(EV value) and the like) for imaging through a user interface 29 in thecamera body 14. The user interface 29 is connected to the camera bodycontroller 28 (the device control unit 34 and the image processing unit35), and various settings determined and changed by the user arereflected in various kinds of processing in the camera body controller28.

Image data subjected to the image processing in the camera bodycontroller 28 is sent to the computer 60 and the like through theinput/output interface 32. The format of image data sent from thedigital camera 10 (camera body controller 28) to the computer 60 and thelike is not particularly limited, and may be an arbitrarily format, suchas RAW, JPEG, or TIFF. Accordingly, the camera body controller 28 mayconstitute a plurality of pieces of associated data, such as headerinformation (imaging information (imaging date and time, model, pixelnumber, diaphragm value, and the like)), main image data, and thumbnailimage data, as one image file in association with one another, like aso-called exchangeable image file format (Exit), and may transmit theimage file to the computer 60.

The computer 60 is connected to the digital camera 10 through theinput/output interface 32 of the camera body 14 and a computerinput/output unit 62, and receives data, such as image data, sent fromthe camera body 14. A computer controller 64 integrally controls thecomputer 60, and subjects image data from the digital camera 10 to imageprocessing or performs communication control with a server 80 or thelike connected to the computer input/output unit 62 through a networkline, such as the Internet 70. The computer 60 has a display 66, and theprocessing content in the computer controller 64 is displayed on thedisplay 66 as necessary. The user operates input means (not shown), suchas a keyboard, while confirming the display of the display 66, therebyinputting data or commands to the computer controller 64. With this, theuser can control the computer 60 or the devices (the digital camera 10and the server 80) connected to the computer 60.

The server 80 has a server input/output unit 82 and a server controller84. The server input/output unit 82 constitutes a transmission/receptionconnection unit with the external devices, such as the computer 60, andis connected to the computer input/output unit 62 of the computer 60through the network line, such as the Internet 70. The server controller84 cooperates with the computer controller 64 according to a controlinstruction signal from the computer 60, performs transmission/receptionof data with the computer controller 64 as necessary, downloads data tothe computer 60, and performs calculation processing and transmits thecalculation result to the computer 60.

Each controller (the lens unit controller 20, the camera body controller28, the computer controller 64, or the server controller 84) includescircuits necessary for control processing, and includes, for example, anarithmetic processing circuit (CPU or the like), a memory, and the like.Communication among the digital camera 10, the computer 60, and theserver 80 may be performed in a wired manner or in a wireless manner.The computer 60 and the server 80 may be constituted integrally, and thecomputer 60 and/or the server 80 may be omitted. A communicationfunction with the server 80 may be provided in the digital camera 10,and transmission/reception of data may be performed directly between thedigital camera 10 and the server 80.

<Point Image Restoration Processing>

Next, point image restoration processing of captured data (image data)of an object image obtained through the imaging element 26 will bedescribed.

In the following example, although an example where the point imagerestoration processing is carried out in the camera body 14 (the camerabody controller 28) will be described, the whole or a part of the pointimage restoration processing can be carried out in another controller(the lens unit controller 20, the computer controller 64, the servercontroller 84, or the like).

The point image restoration processing of this example includesprocessing for subjecting source image data acquired from the imagingelement 26 by capturing the object image using the optical system (thelens 16, the diaphragm 17, or the like) to phase recovery processingusing a phase recovery filter based on a point spread function of theoptical system to acquire recovered image data, and processing forsubjecting source image data to frequency (amplitude) recoveryprocessing using a frequency recovery filter based on the point spreadfunction of the optical system to acquire recovered image data.

FIG. 3 is a diagram showing an outline from image capturing to the pointimage restoration processing.

As shown in FIG. 3, in a case where imaging is performed with a pointimage as an object, an object image is received by the imaging element26 (image sensor) through the optical system (the lens 16, the diaphragm17, or the like), and source image data Do is output from the imagingelement 26. Source image data Do has an amplitude component and a phasecomponent which are deteriorated by a point spread phenomenon resultingfrom the characteristic of the optical system, and the original objectimage (point image) becomes point-asymmetrical blurred image. In FIG. 3,symbol a represents point image information (point spread function) (foreach diaphragm, each focal distance, and each image height) according tothe imaging conditions.

The point image restoration processing of the blurred image isprocessing for determining the characteristic of deterioration (pointspread function (PSF)/optical transfer function (OTF)) according toaberration or the like of the optical system and subjecting a capturedimage (deteriorated image) to restoration processing using a restoration(recovery) filter generated based on the PSF/OTF to restore ahigh-resolution image.

The PSF and the OTF have a relationship of Fourier conversion, the PSFis a real function, and the OTF is a complex function. As a functionhaving information equivalent to the PSF and the OTF, a modulationtransfer function or an amplitude transfer function (MTF) and a phasetransfer function (PTF) are known and respectively indicate an amplitudecomponent and a phase component of the OTF. The MTF and the PTF arecombined to have the amount of information equivalent to the OTF or thePSF.

In general, in the restoration of the blurred image by the PSF, aconvolution type Wiener filter can be used. A frequency characteristicd(ω_(x),ω_(y)) of the restoration filter can be calculated by thefollowing expression with reference to the OTF obtained byFourier-transforming PSF(x,y) and information of a signal to noise ratio(SNR).

$\begin{matrix}{{d\left( {\omega_{x},\omega_{y}} \right)} = \frac{H^{*}\left( {\omega_{x},\omega_{y}} \right)}{{{H\left( {\omega_{x},\omega_{y}} \right)}}^{2} + {1/{{SNR}\left( {\omega_{x},\omega_{y}} \right)}}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In this expression, H(ω_(x),ω_(y)) represents the OTF, andH*(ω_(x),ω_(y)) represents a complex conjugate thereof. Furthermore,SNR(ω_(x),ω_(y)) represents the SN ratio.

Design for the filter coefficient of the restoration filter is anoptimization problem of selecting a coefficient value such that thefrequency characteristic of the filter becomes closest to a desiredWiener frequency characteristic, and the filter coefficient isappropriately calculated by an arbitrary known method.

In this example, a phase recovery filter F1 for recovering deteriorationof a phase characteristic is calculated by using the PTF indicating thephase component of the OTF instead of the OTF of the expression of[Equation 1], calculating the frequency characteristic of the filter,and selecting the coefficient value such that the calculated frequencycharacteristic of the filter becomes closest to the desired Wienerfrequency characteristic. Similarly, a frequency recovery filter F2 forrecovering deterioration of a frequency characteristic is calculated byusing the MTF indicating the amplitude component of the OTF instead ofthe OTF of the expression of [Equation 1], calculating the frequencycharacteristic of the filter, and selecting the coefficient value suchthat the calculated frequency characteristic of the filter becomesclosest to the desired Wiener frequency characteristic.

As shown in FIG. 3, in order to restore the original object image (pointimage) from source image data Do of the blurred image, source image dataDo is subjected to phase recovery processing P10 using the phaserecovery filter F1. A point-asymmetrical image is moved in afrequency-dependent manner and is recovered to a point-symmetrical imageby the phase recovery processing P10.

Subsequently, image data after the phase recovery processing issubjected to nonlinear gradation correction processing P12 (gammacorrection processing by logarithmic processing). The gradation (gamma)correction processing is processing for correcting image datanonlinearly such that an image is naturally reproduced by a displaydevice.

Next, image data after the gradation correction processing is subjectedto frequency recovery processing P14 using the frequency recovery filterF2. The point-symmetrical blurred image is frequency-recovered andbecomes small by the frequency recovery processing P14. With this,recovered image data Dr representing an image (recovered image) closerto the original object image (point image) is obtained.

The phase recovery filter F1 used in the phase recovery processing P10is obtained from point image information (PSF, OTF, or PTF) of theoptical system according to the imaging conditions at the time ofacquisition of source image data Do by a predetermined phase recoveryfilter calculation algorithm P20, and the frequency recovery filter F2used in the frequency recovery processing P14 is obtained from the pointimage information (PSF, OTF, or MTF) of the optical system according tothe imaging conditions at the time of acquisition of source image dataDo by a predetermined frequency recovery filter calculation algorithmP22.

Since the point image information of the optical system fluctuatesdepending on various imaging conditions, such as a diaphragm amount, afocal distance, a zoom amount, an image height, a recording pixelnumber, and a pixel pitch, as well as the type of the lens 16, whencalculating the phase recovery filter F1 and the frequency recoveryfilter F2, these imaging conditions are acquired.

Each of the phase recovery filter F1 and the frequency recovery filterF2 is a filter on an actual space constituted of, for example, N×M(where N and M are integers equal to or greater than two) taps, and isapplied to image data to be processed. With this, weighted averagecalculation (deconvolution calculation) of a filter coefficientallocated to each tap and corresponding pixel data (pixel data to beprocessed of image data and adjacent pixel data) is performed, wherebypixel data after the recovery processing can be calculated. The weightedaveraging processing using the phase recovery filter F1 and thefrequency recovery filter F2 is applied to all pixel data constitutingimage data while changing a target pixel in order, thereby performingthe point image restoration processing.

The phase recovery filter F1 or the frequency recovery filter F2 on theactual space constituted of N×M taps can be derived by inverselyFourier-transforming a phase characteristic of a recovery filter or afrequency amplitude characteristic of a recovery filter on a frequencyspace. Accordingly, the phase recovery filter F1 or the frequencyrecovery filter F2 on the actual space can be appropriately calculatedby specifying a phase recovery filter or a frequency recovery filter ona frequency space as the basis and designating the number of tapsconstituting the phase recovery filter F1 or the frequency recoveryfilter F2 on the actual space. It is preferable that, in order toperform phase correction with excellent accuracy, the number of N×M tapsof the phase recovery filter F1 is greater than the number of taps ofthe frequency recovery filter F2.

Next, an adverse effect in image quality caused by the point imagerestoration processing will be described.

FIG. 4 is a diagram showing an example of a change in image quality ofan edge portion (image boundary portion) in an object image, and shows acase where ideal point image restoration processing (no saturation of apixel value and no clipping) is performed for phase-recovering andfrequency-recovering source image data Do before nonlinear gradationcorrection. Reference numeral 1051 of FIG. 4 indicates contrast inherentin the object image, reference numeral 1052 indicates contrast in sourceimage data Do before the point image restoration processing, andreference numeral 1053 indicates contrast in recovered image data Drafter the ideal point image restoration processing. The transversedirection (X direction) of FIG. 4 indicates a position (one-dimensionalposition) in the object image, and the longitudinal direction (Ydirection) indicates strength of contrast.

As described above, “an edge portion having a difference in level ofcontrast” (see reference numeral 1051 of FIG. 4) in the object image hasimage blur in the captured image (source image data Do) due to the pointspread phenomenon of the optical system at the time of imaging (seereference numeral 1052 of FIG. 4), and recovered image data Dr isobtained through the point image restoration processing (see referencenumeral 1053 of FIG. 4).

In the point image restoration processing, in a case where the “actualimage deterioration characteristics (image blur characteristics)” match“the point spread function (PSF or the like) as the basis of therecovery filter to be used”, the image is appropriately restored, andrecovered image data Dr in which the edge portion or the like isappropriately restored can be obtained (see FIG. 4).

However, in actual point image restoration processing, there may be acase where the “actual image deterioration characteristics (image blurcharacteristics)” do not completely match “the point spread functionassumed by the recovery filter to be used”.

FIG. 5 is a diagram showing an example of source image data, recoveredimage data subjected to the point image restoration processing using therestoration filter, and image data after the gradation correctionprocessing in a case where the “actual image deteriorationcharacteristics (image blur characteristics)” do not completely match“the point spread function assumed by the recovery filter to be used”.

The transverse direction (X direction) in FIG. 5 indicates a position(one-dimensional position) in an image, and the longitudinal direction(Y direction) indicates a pixel value. In a case where the “actual imagedeterioration characteristics (image blur characteristics)” do notcompletely match “the point spread function assumed by the recoveryfilter to be used”, overshoot or undershoot may occur in the edgeportion where a contrast difference is comparatively great (seereference numerals 1061 and 1062 of FIG. 5). Even in a case where imagequality deterioration, such as overshoot or undershoot), occurs, as longas the point image restoration processing is excellent in imagereproducibility and image toughness (image invulnerability), recoveredimage data Dr in which image quality is recovered to such an extent thatimage quality deterioration is not visually recognized (inconspicuous)can be acquired.

However, even if recovered image data which has been recovered to suchan extent that image quality deterioration is inconspicuous has beenobtained through the point image restoration processing, image qualitydeterioration in recovered image data may be enhanced and madeconspicuous through other kinds of processing (the gradation correctionprocessing (the gamma correction processing or the like)) which areperformed after the point image restoration processing.

For example, as shown in FIG. 5, even in a case where overshoot orundershoot itself caused by the point image restoration processing issmall and the influence thereof is particularly inconspicuous visually,and if the gradation correction processing (the gamma correctionprocessing) is performed subsequently, overshoot or undershoot may beenhanced unnecessarily (see “E1” and “E2” of reference numeral 1063 ofFIG. 5). In particular, a great gain (amplification factor) is appliedto an overshoot or undershoot portion on a shadow side through thesubsequent gamma correction processing, and the overshoot or undershootportion constitutes a portion which greatly inclines toward a black sidein the image edge portion (see “E2” of reference numeral 1063 of FIG.5). This phenomenon is not limited to the point image restorationprocessing, and is common for a case where overshoot occurs in the edgeportion as a result of subjecting image data in a linear antilogarithmspace to a contour correction processing.

FIG. 6 shows an example of actual point image restoration processing(with saturation of a pixel value and clipping).

As described above, “an edge portion having a difference in level ofcontrast” (see (a) of FIG. 6) in the object image has image blur in thecaptured image (source image data Do) due to the point spread phenomenonof the optical system at the time of imaging (see (b) of FIG. 6), andrecovered image data Dr is obtained by the point image restorationprocessing (see (c) of FIG. 6).

In the point image restoration processing, in a case where the “actualimage deterioration characteristics (image blur characteristics)” match“the point spread function (PSF or the like) assumed by the restorationfilter to be used”, the image is appropriately restored, and recoveredimage data Dr in which the edge portion or the like is appropriatelyrestored can be obtained (see FIG. 4).

However, in source image data including a pixel (saturated pixel) inwhich the pixel value is saturated, an image waveform in a saturatedpixel portion is brought into a clipped state (see FIG. 6). Inparticular, since source image data of an edge portion including asaturated pixel has a waveform close to a step signal (see (b) of FIG.6), change in contrast becomes extremely large which is not possiblewith the assumed point spread function, and as a result, data in whichdeterioration (image blur) is too small is obtained. In this way, insource image data including a saturated pixel, deviation from originalimage data (object image) occurs due to clipping of pixel data. Ifsource image data in which such data deviation occurs is subjected torestoration processing using a restoration filter, ringing is likely tooccur due to excessive enhancement, and occurring ringing is likely tobecome complicated (see (c) of FIG. 6). Furthermore, a high frequencycomponent increases, and wrap-around noise is likely to be enhanced.

Accordingly, in a case of actually designing the point image restorationprocessing as a part of the image processing flow, it is preferable todesign an overall image processing flow in consideration of not only thepoint image restoration processing itself but also the relevance to theprocessing before and after the point image restoration processing.

In a case of performing gradation correction by the logarithmicprocessing (the gamma correction processing), a frequency recoveryfilter itself may have a filter coefficient corresponding to image databefore the logarithmic processing, or may have a filter coefficientcorresponding to image data after the logarithmic processing.

In a case where recovery processing (frequency recovery processing) isperformed by intentionally applying “a frequency recovery filter havinga filter coefficient corresponding to a pixel value (antilogarithm pixeldata) before gradation correction (before the logarithmic processing)”to “a pixel value (logarithm pixel data) of image data after gradationcorrection (after the logarithmic processing)”, toughness for imagequality deterioration (ringing or the like) occurring in a recoveredimage (restored image) can be improved, and ringing can be madeinconspicuous on the recovered image. This is because, in pixel data(image data) after the logarithmic processing, the gradation of the lowbrightness portion is enhanced and the gradation of the high brightnessportion is not enhanced.

FIG. 7 is a diagram (graph) showing an example of the relationshipbetween pre-processing data and post-processing data by the gammaprocessing (the gradation processing in the logarithmic processing). Thehorizontal axis of FIG. 10 indicates pre-processing data (gammaprocessing input data “IN”), the vertical axis indicates post-processingdata (gamma processing output data “OUT”), and a solid line in the graphindicates a gamma processing gradation curve.

In general point image restoration processing for image data, a regionwhere the effect of the point image restoration processing is easilyvisually recognized is a region with low contrast, and is “a region witha comparatively small level difference of the pixel value” to beapproximated linearly in the gamma processing gradation curve (see “A”of FIG. 7). Meanwhile, in a region with high contrast, that is, in “aregion with a comparatively great level difference of the pixel value”constituting a curved portion in the gamma processing gradation curve,original contrast is high and blur is hardly recognized (see “B” of FIG.7).

In addition, in a region including a saturated pixel out of the regionwith high contrast, if the point image restoration processing (frequencyrecovery processing) is performed for pixel data (pixel data beforegradation correction) whose pixel value is antilogarithm, and thengradation correction (the gamma correction processing, that is, thegradation processing in the logarithmic processing) is performed,undershoot/overshoot (ringing) is likely to be conspicuous. In contrast,in a case where the frequency recovery processing is performed for pixeldata after the logarithmic processing, high contrast is compressedthrough the logarithmic processing, and intensity of ringing due to thefrequency recovery processing is reduced.

That is, the recovery processing (frequency recovery processing) isperformed for pixel data after the logarithmic processing using thefrequency recovery filter having the filter coefficient corresponding topixel data whose pixel value is antilogarithm, whereby it is possible tocarry out the frequency recovery processing for the low contrast region,which is generally likely to be visually recognized, with nodeterioration, and it is possible to reduce the degree of enhancement ofringing in the high contrast region where ringing is likely to occur dueto the frequency recovery processing.

In particular, in a case where the image processing device (imagingdevice or the like) can execute a plurality of kinds of gradationcorrection (gamma correction processing) and stores data of a pluralityof kinds of gamma processing gradation curves, in the related art (seeJP2013-020610A), it is necessary to calculate the limit value of thevariation of the pixel signal value for each of a plurality of types ofgradation correction. However, according to this system, since thefrequency recovery processing is applied to pixel data after gradationcorrection, switching of the processing according to the type ofgradation correction is not required.

In general, the point spread function (PSF) is based on the assumptionthat the input is linear, and a restoration filter based on theassumption is easily generated using a “linear coefficient”, that is, “afilter coefficient corresponding to antilogarithm pixel data”.

In this way, this system is very effective and useful for practical use.

Meanwhile, the recovery processing (frequency recovery processing) isperformed for the pixel value (logarithm pixel data) after gradationcorrection (the logarithmic processing) using the frequency recoveryfilter having the filter coefficient corresponding to the pixel value(logarithm pixel data) after the logarithmic processing, whereby it ispossible to improve toughness with respect to image qualitydeterioration due to ringing caused by the frequency recoveryprocessing, and to make caused ringing inconspicuous on the image.

That is, in a case where pixel data has the pixel value (logarithm pixeldata) after gradation correction (the logarithmic processing), thefrequency recovery processing is performed using the frequency recoveryfilter having the filter coefficient corresponding to the pixel value(logarithm pixel data) after the logarithmic processing, whereby it ispossible to accurately perform the frequency recovery processing itself.In this case, target image data of the frequency recovery processing isset to “source image data after gradation correction”, high contrast iscompressed through gradation correction (the logarithmic processing),and it is possible to reduce strength of ringing caused by the frequencyrecovery processing.

The frequency recovery filter for use in the frequency recoveryprocessing may be generated in advance, or may be successivelycalculated and generated according to the execution of the frequencyrecovery processing. From the viewpoint of reducing the calculationamount during the frequency recovery processing, it is preferable thatthe frequency recovery filter is generated in advance. Furthermore, fromthe viewpoint of using the frequency recovery filter excellent inadaptability, it is preferable that the frequency recovery filter issuccessively calculated at the time of the execution of the frequencyrecovery restoration processing.

In a case where the frequency recovery filter is generated in advance,the filter coefficient of the frequency recovery filter may bedetermined by performing calculation based on the pixel value determinedthrough the logarithmic processing (the gamma correction processing) forthe input pixel value (input image data). The pixel value which is usedfor generating the frequency recovery filter may be a brightness valueor a pixel value (for example, the pixel value of G) relating to onechannel representatively selected among RGB color data. Furthermore, thepixel value which is used for generating the frequency recovery filtermay be a pixel value of a main object, or may be a pixel value which isdetermined from the average value of the entire image.

First Embodiment

FIG. 8 is a block diagram showing a first embodiment of an imageprocessing unit 35 (camera body controller 28) as an image processingdevice according to the invention.

The image processing unit 35 of the first embodiment comprises an offsetcorrection processing unit 41, a WB correction processing unit 42 whichadjusts white balance (WB), a demosaic processing unit 43, a phaserecovery processing unit 44, a gradation correction processing unit 45including a gamma correction processing unit, a frequency recoveryprocessing unit 46, and a brightness/color difference conversionprocessing unit 47 corresponding to a form of a brightness datageneration unit. The phase recovery processing unit 44 and the frequencyrecovery processing unit 46 correspond to the point image restorationcontrol processing unit 36 in the image processing unit 35 shown in FIG.2.

In FIG. 8, the offset correction processing unit 41 receives mosaic data(RAW data: color data (RGB data) in a mosaic pattern of red (R), green(G), and blue (B)) before image processing acquired from the imagingelement 26 in a dot sequence as input. Mosaic data is, for example, data(data having two bytes per pixel) having the bit length of 12 bits (0 to4095) for each of RGB.

The offset correction processing unit 41 is a processing unit whichcorrects a dark current component included in input mosaic data, andperforms offset correction of mosaic data by subtracting the signalvalue of optical black (OB) obtained from a light shielding pixel on theimaging element 26 from mosaic data.

Mosaic data (RGB data) subjected to the offset correction is applied tothe WB correction processing unit 42. The WB correction processing unit42 multiplies RGB data by a WB gain set for each color of RGB andperforms white balance correction of RGB data. In regard to the WB gain,for example, a light source type is automatically determined based onRGB data or a light source type is manually selected, and the WB gainsuitable for the determined or selected light source type is set;however, the setting method of the WB gain is not limited thereto, andthe WB gain can be set by other known methods.

The demosaic processing unit 43 is a unit which performs demosaicprocessing (also referred to as “synchronization processing”) tocalculate all kinds of color information for each pixel from a demosaicimage corresponding to a color filter array of the single plate typeimaging element 26, and for example, in a case of an imaging elementhaving color filters of three colors of RGB, calculates colorinformation of all of RGB for each pixel from a mosaic image made ofRGB. That is, the demosaic processing unit 43 generates image data ofthree phases of RGB synchronized from mosaic data (RGB data in a dotsequence).

RGB data subjected to the demosaic processing is applied to the phaserecovery processing unit 44, and the phase recovery processing of RGBdata is performed in the phase recovery processing unit 44.

FIG. 9 is a block diagram showing an embodiment of the phase recoveryprocessing unit 44.

The phase recovery processing unit 44 has a phase recovery calculationprocessing unit 44 a, a filter selection unit 44 b, an optical systemdata acquisition unit 44 c, and a storage unit 44 d.

The optical system data acquisition unit 44 c acquires optical systemdata indicating the point spread function of the optical system (thelens 16, the diaphragm 17, or the like). Optical system data is datawhich is the selection criterion for a phase recovery filter in thefilter selection unit 44 b, and may be information which directly orindirectly indicates a point spread function of an optical system usedat the time of capturing and acquiring source image data to beprocessed. Accordingly, for example, a transfer function (PSF, OTF (MTF,PTF)) itself relating to the point spread function of the optical systemmay be used as optical system data, or the type of the optical system(for example, the model number or the lens unit 12 (lens 16) or the likeused at the time of imaging) indicating indirectly indicating a transferfunction relating to the point spread function of the optical system, orthe like may be used as optical system data.

The storage unit 44 d stores a phase recovery filter(F_(R1),F_(G1),F_(B1)) of each of RGB generated based on the transferfunctions (PSF, OTF, or PTF) relating to the point spread functions of aplurality of kinds of optical systems. The reason that the phaserecovery filter (F_(R1),F_(G1),F_(B1)) is stored for each of RGB is thatthe aberration of the optical system is different (the PSF shape isdifferent) depending on the wavelength of each color of RGB. It ispreferable that the storage unit 44 d stores the phase recovery filter(F_(R1),F_(G1),F_(B1)) corresponding to F number, a focal distance, animage height, or the like. This is because the PSF shape is differentdepending on these conditions.

The filter selection unit 44 b selects a phase recovery filtercorresponding to optical system data of the optical system used incapturing and acquiring source image data from the phase recoveryfilters stored in the storage unit 44 d based on optical system dataacquired by the optical system data acquisition unit 44 c. The phaserecovery filter (F_(R1),F_(G1),F_(B1)) of each of RGB selected by thefilter selection unit 44 b is sent to the phase recovery calculationprocessing unit 44 a.

While the filter selection unit 44 b ascertains type information (phaserecovery filter storage information) of the phase recovery filter storedin the storage unit 44 d, a method of ascertaining the phase recoveryfilter storage information by the filter selection unit 44 b is notparticularly limited. For example, the filter selection unit 44 b mayhave a storage unit (not shown) which stores the phase recovery filterstorage information, and in a case where the type information of thephase recovery filter stored in the storage unit 44 d is changed, thephase recovery filter storage information stored in the storage unit ofthe filter selection unit 44 b may be changed. The filter selection unit44 b may be connected to the storage unit 44 d and may directlyascertain “information of the phase recovery filter stored in thestorage unit 44 d”, or may ascertain the phase recovery filter storageinformation from a different processing unit (memory or the like) whichascertains the phase recovery filter storage information.

The filter selection unit 44 b may select a phase recovery filtercorresponding to the PSF of the optical system used in capturing andacquiring source image data, and a selection method of the phaserecovery filter is not particularly limited. For example, in a casewhere optical system data from the optical system data acquisition unit44 c directly indicates the PSF, the filter selection unit 44 b selectsa phase recovery filter corresponding to the PSF indicated by opticalsystem data. In a case where optical system data from the optical systemdata acquisition unit 44 c indirectly indicates the PSF, the filterselection unit 44 b selects a phase recovery filter corresponding to thePSF of the optical system used in capturing and acquiring source imagedata from “optical system data indirectly indicating the PSF”.

Source image data (RGB data) subjected to the demosaic processing isinput to the phase recovery calculation processing unit 44 a, and thephase recovery calculation processing unit 44 a subjects RGB data to thephase recovery processing using the phase recovery filter(F_(R1),F_(G1),F_(B1)) selected by the filter selection unit 44 b andcalculates phase-recovered image data. That is, the phase recoverycalculation processing unit 44 a performs deconvolution calculation ofthe phase recovery filter (F_(R1),F_(G1),F_(B1)) and pixel data (pixeldata to be processed and adjacent pixel data) of each of RGBcorresponding to the phase recovery filter and calculates RGB datasubjected to the phase recovery processing.

The phase recovery processing unit 44 configured as above can performthe phase recovery processing reflecting the phase transfer function(PTF) of each color channel of RGB, and can correct various chromaticaberrations, chromatic aberration of magnification and axial chromaticaberration. Since the phase recovery processing unit 44 subjects RGBdata (that is, linear RGB data according to brightness of incident lighton the imaging element 26) before the nonlinear gradation correction tothe phase recovery processing using the phase recovery filtercorresponding to linear data, it is possible to perform accurate phaserecovery.

Returning to FIG. 8, RGB data subjected to the phase recovery processingby the phase recovery processing unit 44 is applied to the gradationcorrection processing unit 45.

The gradation correction processing unit 45 is a unit which subjects RGBdata subjected to the phase recovery processing to the nonlineargradation correction, for example, subjects input RGB data to the gammacorrection processing by the logarithmic processing, and subjects RGBdata to nonlinear processing such that an image is naturally reproducedby a display device.

FIG. 10 is a graph showing an example of input/output characteristics(gamma characteristics) subjected to the gradation correction by thegradation correction processing unit 45. In this example, the gradationcorrection processing unit 45 subjects 12-bit (0 to 4095) RGB data togamma correction corresponding to the gamma characteristics to generate8-bit (0 to 255) color data (1-byte data) of RGB. The gradationcorrection processing unit 45 can be constituted of, for example, alook-up table (LUT) of each of RGB, and preferably subjects each colorof RGB data to the gamma correction. The gradation correction processingunit 45 includes processing for subjecting input data to nonlineargradation correction along a tone curve.

R′G′B′ data subjected to the gradation correction by the gradationcorrection processing unit 45 is applied to the frequency recoveryprocessing unit 46, and the frequency recovery processing of R′G′B datais performed in the frequency recovery processing unit 46.

FIG. 11 is a block diagram showing an embodiment of the frequencyrecovery processing unit 46.

The frequency recovery processing unit 46 has a frequency recoverycalculation processing unit 46 a, a filter selection unit 46 b, anoptical system data acquisition unit 46 c, and a storage unit 46 d.

The filter selection unit 46 b and the optical system data acquisitionunit 46 c respectively correspond to the filter selection unit 44 b andthe optical system data acquisition unit 44 c shown in FIG. 9, and thus,detailed description thereof will not be repeated.

The storage unit 46 d stores a frequency recovery filter(F_(R2),F_(G2),F_(B2)) of each of RGB generated based on the PSF, theOTF, and the MTF of a plurality of kinds of optical systems. The reasonthat the frequency recovery filter (F_(R2),F_(G2),F_(B2)) of each of RGBis stored is that the aberration of the optical system is different (thePSF shape is different) depending on the wavelength of each color ofRGB. It is preferable that the storage unit 46 d stores the frequencyrecovery filter (F_(R2),F_(G2),F_(B2)) corresponding to F number, afocal distance, an image height, or the like. This is because the PSFshape is different depending on these conditions.

The filter selection unit 46 b selects a frequency recovery filtercorresponding to optical system data of the optical system used incapturing and acquiring source image data among the frequency recoveryfilters stored in the storage unit 46 d based on optical system dataacquired by the optical system data acquisition unit 46 c. The frequencyrecovery filter (F_(R2),F_(G2),F_(B2)) of each of RGB selected by thefilter selection unit 46 b is sent to the frequency recovery calculationprocessing unit 46 a.

R′G′B′ data subjected to the gradation correction (gamma correction) isinput to the frequency recovery calculation processing unit 46 a, andthe frequency recovery calculation processing unit 46 a subjects R′G′B′data to the frequency recovery processing using the frequency recoveryfilter (F_(R2),F_(G2),F_(B2)) selected by the filter selection unit 46 band calculates frequency-recovered image data. That is, the frequencyrecovery calculation processing unit 46 a performs deconvolutioncalculation of the frequency recovery filter (F_(R2),F_(G2),F_(B2)) andpixel data (pixel data to be processed and adjacent pixel data) of eachof RGB corresponding to the frequency recovery filter and calculatesR′G′B′ data subjected to the frequency recovery processing.

The frequency recovery processing unit 46 configured as above canperform the frequency recovery processing reflecting the frequencytransfer function (MTF) of each color channel of RGB. Since thefrequency recovery processing is processing after the gradationcorrection, even if slight overshoot/undershoot occurs near the edge dueto the frequency recovery processing, overshoot/undershoot is notamplified by the gradation correction and does not become conspicuousartifact (it is possible to suppress the occurrence of artifact due tothe frequency recovery processing).

R′G′B′ data subjected to the frequency recovery processing by thefrequency recovery processing unit 46 is applied to the brightness/colordifference conversion processing unit 47. The brightness/colordifference conversion processing unit 47 is a processing unit whichconverts R′G′B′ data to brightness data Y′ indicating a brightnesscomponent and color difference data Cr′ and Cb′, and brightness data Y′and color difference data Cr′ and Cb′ can be calculated by the followingexpressions.Y′=0.299R′+0.587G′+0.114B′Cb′=−0.168736R′−0.331264G′+0.5B′Cr′=−0.5R′−0.418688G′−0.081312B′  [Equation 2]

R′G′B′ data is 8-bit data after the gradation correction and thefrequency recovery processing, and the brightness data Y′ and colordifference data Cr′ and Cb′ converted from R′G′B′ data are also 8-bitdata. A conversion expression from R′G′B′ data to brightness data Y′ andcolor difference data Cr′ and Cb′ is not limited to the expression of[Equation 2] described above.

8-bit brightness data Y′ and color difference data Cr′ and Cb′ convertedin this way is subjected to compression processing, such as jointphotographic coding experts group (JPEG), and then, header informationand a plurality of related data, such as compressed main image data andthumbnail image data, are made to correspond to each other andconstituted as a single image file.

The storage unit 44 d (FIG. 8) which stores the phase recovery filtersand the storage unit 46 d (FIG. 11) which stores the frequency recoveryfilters may be provided separately, or may be the same physically andmay have only different storage areas.

In this example, although the phase recovery filters and the frequencyrecovery filters are respectively stored in the storage units 44 d and46 d, and the phase recovery filter and the frequency recovery filterfor use in the recovery processing are appropriately read, the inventionis not limited thereto, and the transfer functions (PSF, OTF, PTF, MTF,and the like) of the optical systems may be stored in the storage unit,the transfer function for use in the recovery processing may be readfrom the storage unit at the time of the recovery processing, and thephase recovery filter and the frequency recovery filter may besequentially calculated.

Second Embodiment

FIG. 12 is a block diagram showing a second embodiment of an imageprocessing unit 35 (camera body controller 28) as an image processingdevice according to the invention. In FIG. 12, the portions common forthe first embodiment of the image processing unit 35 shown in FIG. 8 arerepresented by the same reference numerals, and detailed descriptionthereof will not be repeated.

In the second embodiment, primarily, a frequency recovery processingunit 46-2 is different from the frequency recovery processing unit 46 ofthe first embodiment.

That is, there is a difference in that, while the frequency recoveryprocessing unit 46 of the first embodiment is provided in the post-stageof the gradation correction processing unit 45 and subjects R′G′B′ dataafter the gradation correction to the frequency recovery processing, thefrequency recovery processing unit 46-2 of the second embodiment isprovided in the post-stage of the brightness/color difference conversionprocessing unit 47 and subjects brightness data Y′ (after the gradationcorrection) converted by the brightness/color difference conversionprocessing unit 47 to the frequency recovery processing.

FIG. 13 is a block diagram showing another embodiment of a frequencyrecovery processing unit.

The frequency recovery processing unit 46-2 shown in FIG. 13 has afrequency recovery calculation processing unit 46-2 a, a filterselection unit 46-2 b, an optical system data acquisition unit 46-2 c,and a storage unit 46-2 d.

The filter selection unit 46-2 b and the optical system data acquisitionunit 46-2 c respectively correspond to the filter selection unit 44 band the optical system data acquisition unit 44 c shown in FIG. 9, andthus, detailed description thereof will not be repeated.

The storage unit 46-2 d stores a frequency recovery filter F_(Y2)corresponding to brightness data generated based on the PSF, OTF, andthe PTF of a plurality of kinds of optical systems.

The frequency recovery filter F_(Y2) corresponding to brightness datacan be calculated based on, for example, MTF by mixing the frequencytransfer functions (MTF_(R), MTF_(G), MTF_(B)) of the respective colorchannels of RGB and calculating the modulation transfer function(MTF_(Y)) corresponding to brightness data. When calculating MTF_(Y), itis preferable to calculate MTF_(R), MTF_(G), and MTF_(B) as a weightedlinear sum. The same factor as a factor when generating brightness dataY′ from R′G′B′ data shown in the expression of [Equation 2] can be usedas a weighting factor, but the invention is not limited thereto.

As another example of the frequency recovery filter F_(Y2) correspondingto brightness data, as shown in [Equation 2], a frequency recoveryfilter F_(G2) corresponding to G′ data most contributing to generationof brightness data Y′ may be used as the frequency recovery filterF_(Y2) as it is. It is preferable that the storage unit 46-2 d storesfrequency recovery filters F_(Y2) corresponding to F number, a focaldistance, an image height, and the like.

The filter selection unit 46-2 b selects a frequency recovery filtercorresponding to optical system data of the optical system used incapturing and acquiring source image data among the frequency recoveryfilters stored in the storage unit 46-2 d based on optical system dataacquired by the optical system data acquisition unit 46-2 c. Thefrequency recovery filter F_(Y2) corresponding to brightness dataselected by the filter selection unit 46-2 b is sent to the frequencyrecovery calculation processing unit 46-2 a.

Brightness data Y′ after the gradation correction (gamma correction) isinput to the frequency recovery calculation processing unit 46-2 a, andthe frequency recovery calculation processing unit 46-2 a subjectsbrightness data Y′ to the frequency recovery processing using thefrequency recovery filter F_(Y2) selected by the filter selection unit46-2 b. That is, the frequency recovery calculation processing unit 46-2a performs deconvolution calculation of the frequency recovery filterF_(Y2) and brightness data Y′ (brightness data Y′ of a pixel to beprocessed and adjacent pixels) corresponding to the frequency recoveryfilter F_(Y2) and calculates brightness data Y′ after the frequencyrecovery processing.

The frequency recovery processing unit 46-2 configured as above cansubject brightness data Y′ to the frequency recovery processingreflecting the modulation transfer function (MTF) of brightness data Y′.

Although processing systems for three channels (3ch) are required in thefrequency recovery processing for RGB data by the frequency recoveryprocessing unit 46 of the first embodiment, since a processing systemfor one channel (1ch) is sufficient in the frequency recovery processingfor brightness data Y′, in the frequency recovery processing forbrightness data, it is possible to reduce a circuit scale andcomputational costs, and to reduce the number of frequency recoveryfilters stored in the storage unit 46-2 d.

In the frequency recovery processing for RGB data, if color data of eachcolor of RGB is acquired as assumed (as point spread functioninformation of the optical system), effective frequency recoveryprocessing of RGB data is possible; however, in a case where the actualbehavior of an input signal is not as assumed, in the frequency recoveryprocessing for RGB data, side effects, such as an increase in the numberof places where unnecessary coloring occurs and a conspicuous unnaturaltone of color, may occur.

In contrast, since the frequency recovery processing unit 46-2 of thesecond embodiment performs the frequency recovery processing only forbrightness data, the side effects described above hardly occur (colorsystem toughness in the degree of coloring, the degree of blurring, orthe like).

Since the frequency restoration processing by the frequency recoveryprocessing unit 46-2 is performed for brightness data Y′ after gradationcorrection (gamma correction), similarly to the frequency recoveryprocessing unit 46 of the first embodiment, it is possible to preventoccurred artifact from being enhanced by the gradation correction.

Third Embodiment

FIG. 14 is a block diagram showing a third embodiment of an imageprocessing unit 35 (camera body controller 28) as an image processingdevice according to the invention. In FIG. 14, the portions common forthe first and second embodiments of the image processing unit 35 shownin FIGS. 8 and 12 are represented by the same reference numerals, anddetailed description thereof will not be repeated.

In the third embodiment, primarily, the phase recovery processing unit44-2 is different from the phase recovery processing unit 44 of thefirst and second embodiments.

That is, there is a difference in that, while the phase recoveryprocessing unit 44 of the first and second embodiments is provided inthe post-stage of the demosaic processing unit 43 and subjects demosaicdata of R, G, and B to the phase recovery processing, the phase recoveryprocessing unit 44-2 of the third embodiment is provided in thepost-stage of the brightness/color difference conversion processing unit47 and subjects brightness data Y (before the gradation correction)converted by the brightness/color difference conversion processing unit47 to the phase recovery processing.

FIG. 15 is a block diagram showing another embodiment of phase recoveryprocessing unit.

The phase recovery processing unit 44-2 shown in FIG. 15 has a phaserecovery calculation processing unit 44-2 a, a filter selection unit44-2 b, an optical system data acquisition unit 44-2 c, and a storageunit 44-2 d.

The filter selection unit 44-2 b and the optical system data acquisitionunit 44-2 c respectively correspond to the filter selection unit 44 band the optical system data acquisition unit 44 c shown in FIG. 9, andthus, detailed description thereof will not be repeated.

The storage unit 44-2 d stores a phase recovery filter F_(Y1)corresponding to image data (hereinafter, referred to as “brightnessdata Y”) indicating a brightness component generated based on the PSF,the OTF, and the PTF of a plurality of kinds of optical systems.

The phase recovery filter F_(Y1) corresponding to brightness data Y canbe generated based on PTF calculated by mixing the phase transferfunctions (PTF_(R), PTF_(G), PTF_(B)) of the respective color channelsof RGB and calculating the phase transfer function (PTF_(Y))corresponding to brightness data Y. When calculating PTF_(Y), it ispreferable to calculate PTF_(R), PTF_(G), and PTF_(B) as a weightedlinear sum. The same factor as a factor when generating brightness dataY′ from R′G′B′ data shown in the expression of [Equation 2] can be usedas a weighting factor, but the invention is not limited thereto.

As another example of the phase recovery filter F_(Y1) corresponding tobrightness data Y, as shown in the expression of [Equation 2], a phaserecovery filter F_(G1) corresponding to color data of G mostcontributing to generation of brightness data may be used as the phaserecovery filter F_(Y1) as it is. It is preferable that the storage unit44-2 d stores the phase recovery filters F_(Y1) corresponding to Fnumber, a focal distance, an image height, and the like.

The filter selection unit 44-2 b selects a phase recovery filtercorresponding to optical system data of the optical system used incapturing and acquiring source image data among the phase recoveryfilters stored in the storage unit 44-2 d based on optical system dataacquired by the optical system data acquisition unit 44-2 c. The phaserecovery filter F_(Y1) corresponding to brightness data Y selected bythe filter selection unit 44-2 b is sent to the phase recoverycalculation processing unit 44-2 a.

Brightness data Y before the gradation correction (gamma correction) isinput from the brightness/color difference conversion processing unit 47to the phase recovery calculation processing unit 44-2 a, and the phaserecovery calculation processing unit 44-2 a subjects brightness data Yto the phase recovery processing using the phase recovery filter F_(Y1)selected by the filter selection unit 44-2 b. That is, the phaserecovery calculation processing unit 44-2 a performs deconvolutioncalculation of the phase recovery filter F_(Y1) and brightness data Y(brightness data Y of a pixel to be processed and adjacent pixels)corresponding to the phase recovery filter F_(Y1) and calculatesbrightness data Y subjected to the phase recovery processing.

The phase recovery processing unit 44-2 configured as above can subjectbrightness data Y to the phase recovery processing reflecting the phasetransfer function (PTF) of brightness data Y.

Although processing systems for three channels (3ch) are required in thephase recovery processing for RGB data by the phase recovery processingunit 44 of the first and second embodiments, since a processing systemfor one channel (1ch) is sufficient in the phase recovery processing forbrightness data Y, in the phase recovery processing for brightness data,it is possible to reduce a circuit scale and computational costs, and toreduce the number of phase recovery filters stored in the storage unit44-2 d.

In regard to the phase recovery processing for RGB data, if RGB data isacquired as assumed (as point spread function information of the opticalsystem), effective phase recovery processing of RGB data is possible,and it is possible to effectively reduce chromatic aberration or thelike compared to the phase recovery processing for brightness data;however, in a case where the actual behavior of an input signal is notas assumed, in the phase recovery processing for RGB data, side effects,such as an increase in the number of places where unnecessary coloringoccurs and a conspicuous unnatural tone of color, may occur.

In contrast, since the phase recovery processing unit 44-2 of the thirdembodiment performs the phase recovery processing only for brightnessdata, the side effects described above hardly occur (color systemtoughness in the degree of coloring, the degree of blurring, or thelike).

Similarly to the second embodiment, since the frequency recoveryprocessing by the frequency recovery processing unit 46-2 is performedfor brightness data Y′ after the gradation correction (gammacorrection), the same effects as in the second embodiment are obtained.

Since the image processing unit 35 (point image restoration controlprocessing unit 36) of the third embodiment subjects brightness data Ybefore the gradation correction to the phase recovery processing andsubjects brightness data Y′ after the gradation correction to thefrequency recovery processing, it is possible to minimize a circuitscale and computational costs among the first to third embodiments.

In the third embodiment shown in FIG. 14, the brightness/colordifference conversion processing unit 47 converts respective pieces ofcolor data (RGB) before the gradation correction to brightness data Yand color difference data Cr and Cb, and is different from the first andsecond embodiments in which R′G′B′ data after the gradation correctionis converted to brightness data Y′ and color difference data Cr′ andCb′; however, the processing content is the same.

The gradation correction processing unit 45-2 of the third embodiment isdifferent from the gradation correction processing unit 45 of the firstand second embodiments in that, while the gradation correctionprocessing unit 45 of the first and second embodiments subjects RGB datato the gradation correction (gamma correction), the gradation correctionprocessing unit 45-2 of the third embodiment subjects brightness data Ysubjected to the phase recovery processing by the phase recoveryprocessing unit 44-2 and color difference data Cr and Cb converted bythe brightness/color difference conversion processing unit 47 tononlinear gradation correction (gamma correction). While brightness dataY and color difference data Cr and Cb input to the gradation correctionprocessing unit 45-2 are respectively 12-bit data (2-byte data),brightness data Y′ and color difference data Cr′ and Cb′ after thegradation correction are respectively converted to 8-bit data (1-bytedata).

Modification Examples

The digital camera 10 described above is just for illustration, and theinvention can be applied to other configurations. Each functionalconfiguration can be appropriately realized by arbitrary hardware,software, or a combination thereof. For example, the invention can beapplied to an image processing program which causes a computer toexecute an image processing method (step, processing procedure) in eachdevice and each processing unit (the camera body controller 28, theimage processing unit 35, the gradation correction processing unit 33,the point image restoration control processing unit 36, and the like), acomputer-readable recording medium (non-transitory recording medium)having the image processing program recorded thereon, or variouscomputers on which the image processing program is installable.

<Application Example to EDoF System>

Although the point image restoration processing (the phase recoveryprocessing and the frequency recovery processing) in the embodimentsdescribed above is image processing for performing the frequencyrecovery processing and the phase recovery processing of point spread(point image blur) according to specific imaging condition information(for example, F number, a focal distance, a lens type, and the like) torestore an original object image, image processing to which theinvention can be applied is not limited to the restoration processing inthe embodiments described above. For example, the restoration processingaccording to the invention can also be applied to restoration processingon image data captured and acquired by an optical system (an imaginglens or the like) having an extended depth of field (focus) (EDoF).Image data of a blurred image captured and acquired in a state where thedepth of field (depth of focus) is extended by the EDoF optical systemis subjected to the restoration processing, whereby image data of highresolution in a focused state over a wide range can be restored andgenerated. In this case, the restoration processing is performed using aphase recovery filter and a frequency recovery filter which are based ona transfer function (PSF, OTF, MTF, PTF, or the like) of the EDoFoptical system and have filter coefficients set such that satisfactoryimage restoration can be performed within a range of an extended depthof field (depth of focus).

FIG. 16 is a block diagram showing a form of an imaging module 101including an EDoF optical system. The imaging module (a camera headmounted in a digital camera or the like) 101 of this example includes anEDoF optical system (lens unit) 110, an imaging element 112, and an ADconversion unit 114.

FIG. 17 is a diagram showing an example of an EDoF optical system 110.The EDoF optical system 110 of this example has a single-focus fixedimaging lens 110A, and an optical filter 111 which is arranged at apupil position. The optical filter 111 modulates a phase, and makes theimaging lens 110A constituting the EDoF optical system 110 have anextended depth of field such that an extended depth of field (depth offocus) (EDoF) is obtained. In this way, the imaging lens 110A and theoptical filter 111 constitute a lens unit which modulates a phase toextend a depth of field.

The EDoF optical system 110 includes other components as necessary, andfor example, a diaphragm (not shown) is provided near the optical filter111. The optical filter 111 may be one sheet or may be constituted bycombining a plurality of sheets. The optical filter 111 is only anexample of optical phase modulation means, and the EDoF of the EDoFoptical system 110 (the imaging lens 110A) may be implemented by othermeans. For example, instead of providing the optical filter 111, theEDoF of the EDoF optical system 110 may be implemented by the imaginglens 110A designed to have the same function as the optical filter 111of this example.

That is, the EDoF of the EDoF optical system 110 can be implemented byvarious means for changing the wavefront of imaging on the lightreceiving surface of the imaging element 112. For example, “an opticalelement with a variable thickness”, “an optical element with a variablerefractive index (a refractive index distribution type wavefrontmodulation lens or the like)”, “an optical element with a variablethickness or refractive index coating on the lens surface or the like (awavefront modulation hybrid lens, an optical element formed on the lenssurface as a phase plane, or the like)”, or “a liquid crystal elementcapable of modulating a phase distribution of light (a liquid crystalspatial phase modulation element or the like)” may be used as EDoF meansof the EDoF optical system 110. In this way, the invention can beapplied to not only a case where image formation can be performed to beregularly dispersed by an optical wavefront modulation element (theoptical filter 111 (phase plate)), but also a case where the samedispersed images as the case of using the optical wavefront modulationelement can be formed by the imaging lens 110A itself without using theoptical wavefront modulation element.

The EDoF optical system 110 shown in FIGS. 16 and 17 can be reduced insize since a focus adjustment mechanism which performs focus adjustmentmechanically can be omitted, and can be suitably mounted in acamera-equipped mobile phone or a mobile information terminal portable.

An optical image after passing through the EDoF optical system 110having the EDoF is formed on the imaging element 112 shown in FIG. 16and is converted to an electrical signal here.

The imaging element 112 is constituted of a plurality of pixels arrangedin a matrix by a predetermined pattern array (Bayer array, G stripe RIGfull checkered pattern, X-Trans (Registered Trademark) array, honeycombarray, or the like), and each pixel includes a microlens, a color filter(in this example, an RGB color filter), and a photodiode. An opticalimage incident on the light receiving surface of the imaging element 112through the EDoF optical system 110 is converted to a signal charge inthe amount according to the amount of incident light by each photodiodearranged on the light receiving surface. The signal charge of R, G, andB accumulated in each photodiode is sequentially output as a voltagesignal (image signal) for each pixel.

The analog-to-digital conversion unit (AD conversion unit) 114 convertsthe analog R, G, and B image signals output from the imaging element 112for each pixel to digital RGB image signals. The digital image signalsconverted to the digital image signals by the AD conversion unit 114 areoutput as mosaic data (RAW image data).

The image processing unit (image processing device) 35 shown in thefirst to third embodiments described above is applied to mosaic dataoutput from the imaging module 101, whereby it is possible to generaterecovered image data in a focused state over a wide range.

That is, as indicated by reference numeral 1311, in (a) of FIG. 18, apoint image (optical image) after passing through the EDoF opticalsystem 110 is formed on the imaging element 112 as a large point image(blurred image), but is recovered to a small point image(high-resolution image) through the point image restoration processing(the phase recovery processing and the frequency recovery processing) bythe image processing unit (image processing device) 35 as indicated byreference numeral 1312, in (b) of FIG. 18.

In the respective embodiments described above, although a form in whichthe image processing unit (image processing device) 35 is provided inthe camera body 14 (camera body controller 28) of the digital camera 10has been described, the image processing unit (image processing device)35 may be provided in other devices, such as the computer 60 or theserver 80.

For example, when processing image data in the computer 60, the pointimage restoration processing of image data may be performed by the imageprocessing unit (image processing device) 35 provided in the computer60. In a case where the server 80 comprises the image processing unit(image processing device) 35, for example, image data may be transmittedfrom the digital camera 10 or the computer 60 to the server 80, imagedata may be subjected to the point image restoration processing in theimage processing unit (image processing device) 35 of the server 80, andimage data (recovered image data) after the point image restorationprocessing may be transmitted and provided to a transmission source.

An aspect to which the invention can be applied is not limited to thedigital camera 10, the computer 60, and the server 80, and the inventioncan be applied to mobile devices having an imaging function andfunctions (call handling function, communication function, and othercomputer functions) other than imaging, in addition to cameras havingimaging as a major function. As another aspect to which the inventioncan be applied, for example, mobile phones having a camera function,smartphones, personal digital assistants (PDAs), and portable gamemachines are given. Hereinafter, an example of a smartphone to which theinvention can be applied will be described.

<Configuration of Smartphone>

FIG. 19 shows the appearance of a smartphone 201 which is an embodimentof an imaging device of the invention. The smartphone 201 shown in FIG.19 has a flat plate-like housing 202, and includes, on one surface ofthe housing 202, a display input unit 220 in which a display panel 221as a display unit and an operation panel 222 as an input unit areintegrated. The housing 202 includes a speaker 231, a microphone 232, anoperating unit 240, and a camera unit 241. The configuration of thehousing 202 is not limited thereto, and for example, a configuration inwhich a display unit and an input unit are separated can be used, or aconfiguration in which a folding structure or a slide mechanism isprovided.

FIG. 20 is a block diagram showing the configuration of the smartphone201 shown in FIG. 19. As shown in FIG. 20, the smartphone includes, asmajor components, a wireless communication unit 210, a display inputunit 220, a call handling unit 230, an operating unit 240, a camera unit241, a storage unit 250, an external input/output unit 260, a globalpositioning system (GPS) reception unit 270, a motion sensor unit 280, apower supply unit 290, and a main control unit 200. The smartphone 201has, as a major function, a wireless communication function ofperforming mobile wireless communication through a base station deviceBS and a mobile communication network NW.

The wireless communication unit 210 performs wireless communication withthe base station device BS in the mobile communication network NWaccording to an instruction of the main control unit 200.Transmission/reception of various kinds of file data, such as speechdata or image data, electronic mail data, and the like, or reception ofWeb data, streaming data, or the like is performed using wirelesscommunication.

The display input unit 220 is a so-called touch panel which displaysimages (still image and motion image), character information, or thelike under the control of the main control unit 200 to visually transferinformation to the user and detects a user's operation on the displayedinformation, and includes the display panel 221 and the operation panel222.

The display panel 221 uses a liquid crystal display (LCD) or an organicelectro-luminescence display (OELD) as a display device. The operationpanel 222 is a device which is placed such that an image displayed onthe display surface of the display panel 221 is visible, and detects oneor a plurality of coordinates operated with the finger of the user or astylus. If the device is operated with the finger of the user or thestylus, a detection signal generated due to the operation is output tothe main control unit 200. Next, the main control unit 200 detects theoperation position (coordinates) on the display panel 221 based on thereceived detection signal.

As shown in FIG. 19, the display panel 221 and the operation panel 222of the smartphone 201 illustrated as an embodiment of an imaging deviceof the invention are integrated to constitute the display input unit220, and the operation panel 222 is arranged so as to completely coverthe display panel 221. In a case where this arrangement is used, theoperation panel 222 may have a function of detecting a user's operationin an area outside the display panel 221. In other words, the operationpanel 222 may include a detection area (hereinafter, referred to as adisplay area) for a superimposed portion overlapping the display panel221 and a detection area (hereinafter, referred to as a non-displayarea) for an outer edge portion not overlapping the display panel 221.

Although the size of the display area may completely match the size ofthe display panel 221, both do not necessarily match each other. Theoperation panel 222 may include two sensitive areas of an outer edgeportion and an inside portion. In addition, the width of the outer edgeportion is appropriately designed according to the size of the housing202 or the like. Furthermore, as a position detection system which isused in the operation panel 222, a matrix switch system, a resistivefilm system, a surface acoustic wave system, an infrared system, anelectromagnetic induction system, an electrostatic capacitance system,or the like is given, and any system can be used.

The call handling unit 230 includes a speaker 231 and a microphone 232,converts speech of the user input through the microphone 232 to speechdata processable in the main control unit 200 and outputs speech data tothe main control unit 200, or decodes speech data received by thewireless communication unit 210 or the external input/output unit 260and outputs speech from the speaker 231. As shown in FIG. 19, forexample, the speaker 231 and the microphone 232 can be mounted on thesame surface as the surface on which the display input unit 220 isprovided.

The operating unit 240 is a hardware key, such as a key switch, andreceives an instruction from the user. For example, as shown in FIG. 19,the operating unit 240 is a push button-type switch which is mounted onthe side surface of the housing 202 of the smartphone 201, and is turnedon when pressed with a finger or the like and is turned off by arestoration force of the panel or the like if the finger is released.

The storage unit 250 stores a control program or control data of themain control unit 200, application software, address data in associationwith the name, telephone number, and the like of a communicationpartner, data of transmitted and received electronic mail, Web datadownloaded by Web browsing, downloaded content data, or temporarilystores streaming data or the like. The storage unit 250 is constitutedof an internal storage unit 251 embedded in the smartphone and anexternal storage unit 252 which has a detachable external memory slot.The internal storage unit 251 and the external storage unit 252constituting the storage unit 250 are implemented using a memory (forexample, MicroSD (Registered Trademark) memory or the like) of a flashmemory type, a hard disk type, a multimedia card micro type, or a cardtype, or a storage medium, such as a random access memory (RAM) or aread only memory (ROM).

The external input/output unit 260 plays a role of an interface with allexternal devices connected to the smartphone 201, and is provided fordirect or indirect connection to other external devices by communicationor the like (for example, universal serial bus (USB), IEEE1394 or thelike), or a network (for example, Internet, wireless LAN, Bluetooth(Registered Trademark), radio frequency identification (RFID), infrareddata association (IrDA) (Registered Trademark), ultra wideband (UWB)(Registered Trademark), ZigBee (Registered Trademark), or the like).

The external device connected to the smartphone 201 is, for example, awired or wireless headset, a wired or wireless external charger, a wiredor wireless data port, a memory card connected through a card socket, asubscriber identity module (SIM)/user identity module (UIM) card, anexternal audio-video device connected through an audio-videoinput/output (I/O) terminal, an external audio-video device connected ina wireless manner, a smartphone connected in a wired or wireless manner,a personal computer connected in a wired or wireless manner, a PDAconnected in a wired or wireless manner, an earphone, or the like. Theexternal input/output unit can transfer data transmitted from theexternal devices to the respective components in the smartphone 201, orcan transmit data in the smartphone 201 to the external devices.

The GPS reception unit 270 receives GPS signals transmitted from GPSsatellites ST1 to STn according to an instruction of the main controlunit 200, executes positioning calculation processing based on aplurality of received GPS signals, and detects the position of thesmartphone 201 having latitude, longitude, and altitude. When positionalinformation can be acquired from the wireless communication unit 210 orthe external input/output unit 260 (for example, a wireless LAN), theGPS reception unit 270 may detect the position using the positionalinformation.

The motion sensor unit 280 includes, for example, a three-axisacceleration sensor or the like, and detects physical motion of thesmartphone 201 according to an instruction of the main control unit 200.The moving direction or acceleration of the smartphone 201 can bedetected by detecting physical motion of the smartphone 201. Thedetection result is output to the main control unit 200.

The power supply unit 290 supplies power stored in a battery (not shown)to the respective units of the smartphone 201 according to aninstruction of the main control unit 200.

The main control unit 200 includes a microprocessor, operates accordingto the control program or control data stored in the storage unit 250,and integrally controls the respective units of the smartphone 201. Themain control unit 200 has a mobile communication control function ofcontrolling the respective units of a communication system in order toperform speech communication or data communication through the wirelesscommunication unit 210, and an application processing function.

The application processing function is implemented by the main controlunit 200 operating according to application software stored in thestorage unit 250. The application processing function is, for example,an infrared communication function of controlling the externalinput/output unit 260 to perform data communication with a counterdevice, an electronic mail function of transmitting and receivingelectronic mail, a Web browsing function of browsing Web pages, or thelike.

The main control unit 200 has an image processing function of displayingvideo on the display input unit 220, or the like based on image data(still image or motion image data), such as received data or downloadedstreaming data. The image processing function refers to a function ofthe main control unit 200 decoding image data, subjecting the decodingresult to image processing, and displaying an image on the display inputunit 220.

The main control unit 200 executes display control on the display panel221, and operation detection control for detecting a user's operationthrough the operating unit 240 and the operation panel 222.

With the execution of the display control, the main control unit 200displays an icon for activating application software or a software key,such as a scroll bar, or displays a window for creating electronic mail.The scroll bar refers to a software key for receiving an instruction tomove a display portion of an image which is too large to fit into thedisplay area of the display panel 221.

With the execution of the operation detection control, the main controlunit 200 detects a user's operation through the operating unit 240,receives an operation on the icon or an input of a character string inan entry column of the window through the operation panel 222, orreceives a scroll request of a display image through the scroll bar.

Furthermore, with the execution of the operation detection control, themain control unit 200 has a touch panel control function of determiningwhether an operation position on the operation panel 222 is thesuperimposed portion (display area) overlapping the display panel 221 orthe outer edge portion (non-display area) not overlapping the displaypanel 221, and controlling the sensitive area of the operation panel 222or the display position of the software key.

The main control unit 200 may detect a gesture operation on theoperation panel 222 and may execute a function set in advance accordingto the detected gesture operation. The gesture operation is not aconventional simple touch operation, but means an operation to render atrack with a finger or the like, an operation to simultaneouslydesignate a plurality of positions, or an operation to render a trackfor at least one of a plurality of positions by combining theoperations.

The camera unit 241 is a digital camera which electronically captures animage using an imaging element, such as a complementary metal oxidesemiconductor (CMOS) or a charge-coupled device (CCD). The camera unit241 can convert image data obtained by imaging to compressed image data,such as joint photographic coding experts group (JPEG), and can recordimage data in the storage unit 250 under the control of the main controlunit 200. Furthermore, the camera unit 241 can output image data throughthe external input/output unit 260 or the wireless communication unit210. As shown in FIG. 19, in the smartphone 201, the camera unit 241 ismounted on the same surface of the display input unit 220; however, themounting position of the camera unit 241 is not limited thereto, and thecamera unit 241 may be mounted on the rear surface of the display inputunit 220, or a plurality of camera units 241 may be mounted. In a casewhere a plurality of camera units 241 are mounted, the camera unit 241which is used to capture an image is switched from one to another andcaptures an image alone, or a plurality of camera units 241 aresimultaneously used to capture images.

The camera unit 241 is used for various functions of the smartphone 201.For example, an image acquired by the camera unit 241 can be displayedon the display panel 221, or an image in the camera unit 241 can be usedas one operation input on the operation panel 222. When the GPSreception unit 270 detects the position, the position may be detectedwith reference to an image from the camera unit 241. In addition, theoptical axis direction of the camera unit 241 of the smartphone 201 maybe determined or the current use environment can be determined withreference to an image from the camera unit 241 without using thethree-axis acceleration sensor, or using the three-axis accelerationsensor. Of course, an image from the camera unit 241 may be used inapplication software.

In addition, image data of a still image or a motion image can beattached with positional information acquired by the GPS reception unit270, speech information acquired by the microphone 232, speechinformation (may be text information through speech-text conversion inthe main control unit or the like), posture information acquired by themotion sensor unit 280, or the like and can be recorded in the storageunit 250, or may be output through the external input/output unit 260 orthe wireless communication unit 210.

In the smartphone 201 described above, the respective processing unitsdescribed above in connection with the point image restorationprocessing can be appropriately implemented by, for example, the maincontrol unit 200, the storage unit 250, and the like.

The invention is not limited to the embodiments described above, andvarious modifications can be made without departing from the spirit ofthe invention.

EXPLANATION OF REFERENCES

-   -   10: digital camera    -   12: lens unit    -   14: camera body    -   16: lens    -   17: diaphragm    -   18: optical system operating unit    -   20: lens unit controller    -   22: lens unit input/output unit    -   26: imaging element    -   28: body controller    -   29: user interface    -   30: camera body input/output unit    -   32: input/output interface    -   33: gradation correction processing unit    -   34: device control unit    -   35: image processing unit    -   36: point image restoration control processing unit    -   41: offset correction processing unit    -   42: WB correction processing unit    -   43: demosaic processing unit    -   44, 44-2: phase recovery processing unit    -   44 a, 44-2 a: phase recovery calculation processing unit    -   44 b, 44-2 b, 46 b, 46-2 b: filter selection unit    -   44 c, 44-2 c, 46 c, 46-2 c: optical system data acquisition unit    -   44 d, 44-2 d, 46 d, 46-2 d, 250: storage unit    -   45, 45-2: gradation correction processing unit    -   46, 46-2: frequency recovery processing unit    -   46 a, 46-2 a: frequency recovery calculation processing unit    -   47: brightness/color difference conversion processing unit    -   101: imaging module    -   110: EDoF optical system    -   110A: imaging lens    -   111: optical filter    -   112: imaging element    -   114: AD conversion unit    -   200: main control unit    -   201: smartphone    -   202: housing    -   210: wireless communication unit    -   220: display input unit    -   221: display panel    -   222: operation panel    -   230: call handling unit    -   231: speaker    -   232: microphone    -   240: operating unit    -   241: camera unit    -   251: internal storage unit    -   252: external storage unit    -   260: external input/output unit    -   270: reception unit    -   280: motion sensor unit    -   290: power supply unit

What is claimed is:
 1. An image processing device comprising: a phaserecovery processing circuit which subjects image data acquired from animaging element by capturing an object image using an optical system tophase recovery processing using a phase recovery filter based on a pointspread function of the optical system; a gradation correction processingcircuit which subjects image data subjected to the phase recoveryprocessing to nonlinear gradation correction; and a frequency recoveryprocessing circuit which subjects image data subjected to the gradationcorrection to frequency recovery processing using a frequency recoveryfilter based on the point spread function of the optical system.
 2. Theimage processing device according to claim 1, further comprising: astorage which stores the phase recovery filter and the frequencyrecovery filter, wherein the phase recovery processing circuit reads thephase recovery filter from the storage and uses the phase recoveryfilter in the phase recovery processing, and the frequency recoveryprocessing circuit reads the frequency recovery filter from the storageand uses the frequency recovery filter in the frequency recoveryprocessing.
 3. The image processing device according to claim 1, furthercomprising: a storage which stores the point spread function of theoptical system, an optical transfer function obtained byFourier-transforming the point spread function, or a modulation transferfunction indicating an amplitude component of the optical transferfunction and a phase transfer function indicating a phase component ofthe optical transfer function, wherein the phase recovery processingcircuit reads the point spread function, the optical transfer function,or the phase transfer function from the storage, generates the phaserecovery filter, and uses the generated phase recovery filter in thephase recovery processing, and the frequency recovery processing circuitreads the point spread function, the optical transfer function, or themodulation transfer function from the storage, generates the frequencyrecovery filter, and uses the generated frequency recovery filter in thefrequency recovery processing.
 4. The image processing device accordingto claim 1, wherein the phase recovery processing circuit subjects imagedata acquired from the imaging element, which is image data for eachcolor channel, to phase recovery processing using a phase recoveryfilter, and the frequency recovery processing circuit subjects the imagedata subjected to the gradation correction, which is image data for eachcolor channel, to frequency recovery processing using a frequencyrecovery filter.
 5. The image processing device according to claim 2,wherein the phase recovery processing circuit subjects image dataacquired from the imaging element, which is image data for each colorchannel, to phase recovery processing using a phase recovery filter, andthe frequency recovery processing circuit subjects the image datasubjected to the gradation correction, which is image data for eachcolor channel, to frequency recovery processing using a frequencyrecovery filter.
 6. The image processing device according to claim 3,wherein the phase recovery processing circuit subjects image dataacquired from the imaging element, which is image data for each colorchannel, to phase recovery processing using a phase recovery filter, andthe frequency recovery processing circuit subjects the image datasubjected to the gradation correction, which is image data for eachcolor channel, to frequency recovery processing using a frequencyrecovery filter.
 7. The image processing device according to claim 1,wherein the phase recovery processing circuit subjects image dataacquired from the imaging element, which is image data for each colorchannel, to phase recovery processing using the phase recovery filter,and the frequency recovery processing circuit subjects image datasubjected to gradation correction by the gradation correction processingcircuit, which is image data indicating a brightness component generatedfrom image data for each color channel, to frequency recovery processingusing the frequency recovery filter.
 8. The image processing deviceaccording to claim 2, wherein the phase recovery processing circuitsubjects image data acquired from the imaging element, which is imagedata for each color channel, to phase recovery processing using thephase recovery filter, and the frequency recovery processing circuitsubjects image data subjected to gradation correction by the gradationcorrection processing circuit, which is image data indicating abrightness component generated from image data for each color channel,to frequency recovery processing using the frequency recovery filter. 9.The image processing device according to claim 3, wherein the phaserecovery processing circuit subjects image data acquired from theimaging element, which is image data for each color channel, to phaserecovery processing using the phase recovery filter, and the frequencyrecovery processing circuit subjects image data subjected to gradationcorrection by the gradation correction processing circuit, which isimage data indicating a brightness component generated from image datafor each color channel, to frequency recovery processing using thefrequency recovery filter.
 10. The image processing device according toclaim 1, wherein the phase recovery processing circuit subjects imagedata acquired from the imaging element, which is image data indicating abrightness component generated from image data for each color channel,to phase recovery processing using the phase recovery filter, and thefrequency recovery processing circuit subjects the image data subjectedto the gradation correction, which is image data indicating a brightnesscomponent generated from image data for each color channel, to frequencyrecovery processing using the frequency recovery filter.
 11. The imageprocessing device according to claim 2, wherein the phase recoveryprocessing circuit subjects image data acquired from the imagingelement, which is image data indicating a brightness component generatedfrom image data for each color channel, to phase recovery processingusing the phase recovery filter, and the frequency recovery processingcircuit subjects the image data subjected to the gradation correction,which is image data indicating a brightness component generated fromimage data for each color channel, to frequency recovery processingusing the frequency recovery filter.
 12. The image processing deviceaccording to claim 3, wherein the phase recovery processing circuitsubjects image data acquired from the imaging element, which is imagedata indicating a brightness component generated from image data foreach color channel, to phase recovery processing using the phaserecovery filter, and the frequency recovery processing circuit subjectsthe image data subjected to the gradation correction, which is imagedata indicating a brightness component generated from image data foreach color channel, to frequency recovery processing using the frequencyrecovery filter.
 13. The image processing device according to claim 10,further comprising: a brightness data generation circuit which generatesbrightness data indicating a brightness component from image data foreach color channel acquired from the imaging element, wherein the phaserecovery processing circuit subjects brightness data generated by thebrightness data generation circuit to phase recovery processing usingthe phase recovery filter, the gradation correction processing circuitsubjects the brightness data subjected to the phase recovery processingto nonlinear gradation correction, and the frequency recovery processingcircuit subjects the brightness data subjected to the gradationcorrection to frequency recovery processing using the frequency recoveryfilter.
 14. The image processing device according to claim 11, furthercomprising: a brightness data generation circuit which generatesbrightness data indicating a brightness component from image data foreach color channel acquired from the imaging element, wherein the phaserecovery processing circuit subjects brightness data generated by thebrightness data generation circuit to phase recovery processing usingthe phase recovery filter, the gradation correction processing circuitsubjects the brightness data subjected to the phase recovery processingto nonlinear gradation correction, and the frequency recovery processingcircuit subjects the brightness data subjected to the gradationcorrection to frequency recovery processing using the frequency recoveryfilter.
 15. The image processing device according to claim 1, whereinthe gradation correction processing circuit is a gamma correctionprocessing circuit which subjects the image data to gradation correctionby logarithmic processing.
 16. The image processing device according toclaim 1, wherein the bit length of the image data subjected to phaserecovery processing by the phase recovery processing circuit is greaterthan the bit length of the image data subjected to frequency recoveryprocessing by the frequency recovery processing circuit.
 17. The imageprocessing device according to claim 1, wherein the optical system has alens unit which enlarges a depth of field by modulating a phase.
 18. Animaging device comprising: an imaging element which outputs image databy capturing an object image using an optical system; and the imageprocessing device according to claim
 1. 19. An image processing methodusing the image processing device according to claim 1 comprising: astep of subjecting image data acquired from an imaging element bycapturing an object image using an optical system to phase recoveryprocessing using a phase recovery filter based on a point spreadfunction of the optical system; a step of subjecting image datasubjected to the phase recovery processing to nonlinear gradationcorrection; and a step of subjecting image data subjected to thegradation correction to frequency recovery processing using a frequencyrecovery filter based on the point spread function of the opticalsystem.
 20. A non-transitory computer readable recording medium storingan image processing program which causes a computer to execute: a stepof subjecting image data acquired from an imaging element by capturingan object image using an optical system to phase recovery processingusing a phase recovery filter based on a point spread function of theoptical system; a step of subjecting image data subjected to the phaserecovery processing to nonlinear gradation correction; and a step ofsubjecting image data subjected to the gradation correction to frequencyrecovery processing using a frequency recovery filter based on the pointspread function of the optical system.